Embodiments of the present invention relate to the field of displaying high resolution images on displays with lower resolution, where the displays use a triad arrangement to display the R, G, and B or other components of the image. This triad arrangement is common in direct view LCD displays, for example, and in such an arrangement, a single pixel is composed of 3 side-by-side subpixels. Each subpixel controls only one of the three primaries (i.e., R, G and B) and is, in turn, usually controlled solely by the primaries of the digital image representation. The high-resolution image may be available in memory, or may be available directly from an algorithm (vector graphics, some font designs, and computer graphics).
The most commonly used method for displaying high-resolution images on a lower resolution display is to sample the pixels 2 of the high-resolution image 4 down to the resolution of the low-resolution display 6, as shown in FIG. 1. Then, the R, G, B values of each downsampled color pixel 8 are mapped to the separate R, G, B elements 10, 12 and 14 of each display pixel 16. These R, G, B elements 10, 12 and 14 of a display pixel are also referred to as subpixels. Because the display device does not allow overlapping color elements, the subpixels can only take on one of the three R, G, or B colors, however, the color""s amplitude can be varied throughout the entire greyscale range (e.g., 0-255). The subpixels usually have a 1:3 aspect ratio (width:height), so that the resulting pixel 16 is square. The subsampling/mapping techniques do not consider the fact that the display""s R, G, and B subpixels are spatially displaced; in fact they are assumed to be overlapping in the same manner as they are in the high-resolution image. This type of sampling may be referred to as sub-sampling or traditional sub-sampling.
The pixels of the high-resolution image 4 are shown as three slightly offset stacked squares 8 to indicate their RGB values are associated for the same spatial position (i.e., pixel). One display pixel 16, consisting of one each of the R, G and B subpixels 10, 12 and 14 is shown as part of the lower-resolution triad display 6 in FIG. 1 using dark lines. Other display pixels are shown with lighter gray lines.
In this example, the high-resolution image has 3xc3x97 more resolution than the display (in both horizontal and vertical dimensions). Since this direct subsampling technique causes aliasing artifacts, various methods are used, such as averaging the neighboring unsampled pixels in with the sampled pixel. Note that the common technique of averaging neighboring elements while subsampling is mathematically equal to prefiltering the high resolution image with a rectangular (rect) filter. Also, note that techniques of selecting a different pixel than the leftmost (as shown in this figure) can be considered as a prefiltering that affects only phase. Thus, most of the processing associated with preventing aliasing can be viewed as a filtering operation on the high-resolution image, even if the kernel is applied only at the sampled pixel positions.
An achromatic image, as defined in this specification and claims has no visible color variation. This achromatic condition can occur when an image contains only one layer or color channel, or when an image has multiple layers or color channels, but each color layer is identical thereby yielding a single color image.
It has been realized that the aforementioned technique does not take advantage of potential display resolution. Background information in this area may be accessed by reference to R. Fiegenblatt (1989), xe2x80x9cFull color imaging on amplitude color mosaic displaysxe2x80x9d Proc. SPIE V. 1075, 199-205; and J. Kranz and L. Silverstein (1990) xe2x80x9cColor matrix display image quality: The effects of luminance and spatial samplingxe2x80x9d, SID Symp. Digest 29-32 which are hereby incorporated herein by reference.
For example, in the display shown in FIG. 1, while the display pixel 16 resolution is ⅓ that of the high resolution image (source image) 4, the subpixels 10, 12 and 14 are at a resolution equal to that of the source (in the horizontal dimension). If this display were solely to be used by colorblind individuals, it would be possible to take advantage of the spatial positions of the subpixels. This approach is shown in FIG. 2 below, where the R, G, and B subpixels 10, 12 and 14 of the display are taken from the corresponding colors of different pixels 11, 13 and 15 of the high-resolution image. This allows the horizontal resolution to be at the subpixel resolution, which is 3xc3x97 that of the display pixel resolution.
But what about the viewer of the display who is not color-blind? That is, the majority of viewers. Fortunately for display engineers, even observers with perfect color vision are color blind at the highest spatial frequencies. This is indicated below in FIG. 3, where idealized spatial frequency responses of the human visual system are shown.
Here, luminance 17 refers to the achromatic contact of the viewed image, and chrominance 19 refers to the color content, which is processed by the visual system as isoluminant modulations from red to green, and from blue to yellow. The color difference signals R-Y and B-Y of video are rough approximations to these modulations. For most observers, the bandwidth of the chromatic frequency response is xc2xd that of the luminance frequency response. Sometimes, the bandwidth of the blue-yellow modulation response is even less, down to about ⅓ of the luminance. Sampling which comprises mapping of color elements from different image pixels to the subpixels of a display pixel triad may be referred to as sub-pixel sampling.
With reference to FIG. 4, in the horizontal direction of the display, there is a range of frequencies that lie between the Nyquist of the display pixel 16 (display pixel=triad pixel, giving a triad Nyquist at 0.5 cycles per triad pixel) and the Nyquist frequency of the sub-pixels pixels elements 10, 12 and 14 (0.5 cycles per subpixel=1.5 cycles/triad pixels). This region is shown as the rectangular region 20 in FIG. 4. The resulting sinc functions from convolving the high resolution image with a rect function whose width is equal to the display sample spacing is shown as a light dashed-dot curve 22. This is the most common approach taken for modeling the display MTF (modulation transfer function) when the display is an LCD.
The sinc function resulting from convolving the high-res source image with a rect equal to the subpixel spacing is shown as a dashed curve 24, which has higher bandwidth. This is the limit imposed by the display considering that the subpixels are rect in 1D. In the shown rectangular region 20, the subpixels can display luminance information, but not chromatic information. In fact, any chromatic information in this region is aliased. Thus, in this region, by allowing chromatic aliasing, we can achieve higher frequency luminance information than allowed by the triad (i.e., display) pixels. This is the xe2x80x9cadvantagexe2x80x9d region afforded by using sub-pixel sampling.
For applications with font display, the black and white fonts are typically preprocessed, as shown in FIG. 5. The standard pre-processing includes hinting, which refers to the centering of the font strokes on the center of the pixel, i.e., a font-stroke specific phase shift. This is usually followed by low-pass filtering, also referred to as greyscale antialiasing.
The visual frequency responses (CSFs) shown in FIG. 3 are idealized. In practice, they have a finite falloff slope, as shown in FIG. 6A. The luminance CSF 30 has been mapped from units of cy/deg to the display pixel domain (assuming a viewing distance of 1280 pixels). It is shown as the solid line 30 that has a maximum frequency near 1.5 cy/pixel (display pixel), and is bandpass in shape with a peak near 0.2 cy/pixel triad. The R:G CSF 32 is shown as the dashed line, that is lowpass with a maximum frequency near 0.5 cy/pixel. The B:Y modulation CSF 34 is shown as the dashed-dotted LPF curve with a similar maximum frequency as the R:G CSF, but with lower maximum response. The range between the cutoff frequencies of the chroma CSF 32 and 34 and the luminance CSF 30 is the region where we can allow chromatic aliasing in order to improve luminance bandwidth.
FIG. 6A also shows an idealized image power spectra 36 as a 1/f function, appearing in the figure as a straight line with a slope of xe2x88x921 (since the figure is using log axes). This spectrum will repeat at the sampling frequency. These repeats are shown for the pixel 38 and the subpixel 40 sampling rates for the horizontal direction. The one occurring at lower frequencies 38 is due to the pixel sampling, and the one at the higher frequencies 40 is due to the subpixel sampling. Note that the shapes change since we are plotting on a log frequency axis. The frequencies of these repeat spectra that extend to the lower frequencies below Nyquist are referred to as aliasing. The leftmost one is chromatic aliasing 38 since it is due to the pixel sampling rate, while the luminance aliasing 40 occurs at higher frequencies because it is related to the higher sub-pixel sampling rate.
In FIG. 6A, no prefiltering has been applied to the source spectra. Consequently, aliasing, due to the pixel sampling (i.e., chromatic aliasing), extends to very low frequencies 35. Thus even though the chromatic CSF has a lower bandwidth than the luminance CSF, the color artifacts may still be visible (depending on the noise and contrast of the display).
In FIG. 6B, we have applied the prefilter (a rect function equal to three source image pixels), shown in FIG. 4 as a dashed-dotted line 22, to the source power spectrum, and it can be seen to affect the baseband spectrum 42 past 0.5 cy/pixel, causing it to have a slope steeper than xe2x88x921 shown at 44. The repeats also show the effect of this prefilter. Even with this filter, we see that some chromatic aliasing (the repeated spectrum at the lower frequencies) occurs at frequencies 46 lower than the cut-off frequency of the two chrominance CSFs 32a and 34a. Thus it can be seen that simple luminance prefiltering will have a difficult time removing chromatic aliasing, without removing all the luminance frequencies past 0.5 cy/pix (i.e., the xe2x80x9cadvantagexe2x80x9d region).
Since we are relying on the visual system differences in bandwidth as a function of luminance or chrominance to give us a luminance bandwidth boost in the xe2x80x9cadvantageous regionxe2x80x9d, one possibility is to design the prefiltering based on visual system models as described in C. Betrisey, et al (2000), xe2x80x9cDisplaced filtering for patterned displays,xe2x80x9d SID Symposium digest, 296-299, hereby incorporated herein by reference and illustrated in FIG. 7.
This technique ideally uses different prefilters depending on which color layer, and on which color subpixel the image is being sampled for. Thus there are 9 filters. They were designed using a human visual differences model described in X. Zhang and B. Wandell (1996) xe2x80x9cA spatial extension of CIELab for digital color image reproductionxe2x80x9d, SID Symp. Digest 731-734, incorporated herein by reference and shown in the FIG. 7. This was done offline, assuming the image is always black and white. In the final implementation, rect functions rather than the resulting filters are used in order to save computations. In addition, there is still some residual chromatic error that can be seen because the chromatic aliasing extends down to lower frequencies than the chromatic CSF cutoff (as seen in FIG. 6B).
However, the visual model used does not take into account the masking properties of the visual system which cause the masking of chrominance by luminance when the luminance is at medium to high contrast levels. So, in larger fonts the chromatic artifacts, which lie along the edges of the font, are masked by the high luminance contrast of the font. However, as the font size is reduced the luminance of the font reduces, and then the same chromatic artifacts become very visible (at very small fonts for example, the b/w portion of the font disappears, leaving only a localized color speckle).
Embodiments of the present invention comprise methods and systems that rely less on filtering and its assumptions of linearity and are capable of working on input color images. These embodiments are capable of directly removing low frequency chromatic artifacts after they are caused by sub-pixel sampling. This is achieved by generating a LPF version of the chromatic content of the image which is added to the luminance and chromatic aliasing versions. This is done by making use of color domains other than additive, primary color domains (i.e., RGB) to remove the color artifacts caused by the sub-pixel sampling. In practice, only the lower frequency chromatic artifacts need to be cancelled, since the high frequency ones cannot be seen due to the lower bandwidth of the chromatic CSFs, as shown in FIG. 6A.
The methods and systems of the present invention may be used in obtaining higher resolution luminance signals with no visibility of chromatic aliasing, when the display is viewed no closer than designed specifications. These techniques do not need the assumption that the source image is text, or that the images are achromatic.
Embodiments of the present invention convert a higher-resolution image to a lower-resolution image with reduced errors caused by the sub-sampling processes. When the higher-resolution image is not in a format which allows separation of luminance and chrominance data, the image is converted to such a format. Many opponent color domains are acceptable. The opponent color domain image is split thereby separating the luminance channel from the chrominance channels thereby allowing for separate processing.
The luminance channel is then converted to an additive color domain (ACD), such as RGB, and the ACD luminance image is sub-pixel sampled to preserve luminance data while reducing resolution. Following sub-pixel sampling, the sub-pixel sampled (SPS) image is converted back to an opponent color domain (OCD) and again split into separate luminance and chrominance channels. The SPS chrominance channels produced by this split are then high-pass filtered to remove low-frequency artifacts produced during sub-pixel sampling. The SPS luminance channel is typically not modified to preserve original luminance data.
The chrominance channels from the original image are low-pass filtered and then sub-sampled to provide the chrominance data for the lower-resolution image. These low-pass filtered chrominance channels are then combined with the high-pass filtered, sub-pixel sampled chrominance channels created from the original luminance channel. These combined chrominance channels are also combined with the SPS luminance channel to form a reduced-error, lower-resolution image, generally in an opponent color domain. This error-reduced, lower-resolution image may then be converted to an additive color domain or some other color domain compatible with the desired application.